Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy

This study critically analyses the information-theoretic, axiomatic and combinatorial philosophical bases of the entropy and cross-entropy concepts. The combinatorial basis is shown to be the most fundamental (most primitive) of these three bases, since it gives (i) a derivation for the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization of a system; and (iii) new, generalized definitions of entropy and cross-entropy supersets of the Boltzmann principle applicable to non-multinomial systems. The combinatorial basis is therefore of much broader scope, with far greater power of application, than the information-theoretic and axiomatic bases. The generalized definitions underpin a new discipline of “combinatorial information theory”, for the analysis of probabilistic systems of any type. Jaynes’ generic formulation of statistical mechanics for multinomial systems is re-examined in light of the combinatorial approach, including the analysis of probability distributions, ensemble theory, Jaynes relations, fluctuation theory and the entropy concentration theorem. Several new concepts are outlined, including a generalized Clausius inequality, a generalized free energy (“free information”) function, and a generalized Gibbs-Duhem relation and phase rule. For nonmultinomial systems, the generalized approach provides a different framework for the reinterpretation of the many alternative entropy measures (e.g. Bose-Einstein, Fermi-Dirac, Rényi, Tsallis, Sharma-Mittal, Beck-Cohen, Kaniadakis) in terms of their combinatorial structure. A connection between the combinatorial and Bayesian approaches is also explored. PACS numbers: 02.50.Cw, 02.50.Tt, 05.20.-y, 05.40.-a, 05.70.-a, 05.70.Ce, 05.90.+m, 89.20.-a, 89.70.+c,